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ABSTRACT 

The Center for Educational Research and Evaluation of 
the University of North Carolina (Greensboro) has been contracted to 
propose standards, indicators of quality, data collection procedures, 
and instruments for the evaluati^on of Georgia public schools mandated 
by the state legislature. The development of these standards and 
indicators is discussed • A comprehensive list of standards and 
indicators was compiled through reviews of literature on educational 
indicators and effective schools foi 10 factors: (1) autonomy of 
school-site management; (2) instructional leadership; (3) staff 
stability; (4) parental involveiri?nt and support; (5) schoolwide 
recognition of academic success; (6) maximized learning time; (7) 
collaborative planning and collegial relationships; (8) sense of 
community; (9) clear grades and high expectations; and (10) order and 
discipline. The initial list was subjected to extensive review and 
revision. The district review panels included: a Technical Advisory 
Board (TAB) of five national experts in educational evaluation; a 
departmental TAB of professional educators from Georgia; an advisory 
committee of 40 Georgia educators and citizens; and a steering 
committee of 15 members of the preceding group. Appendix A describes 
the literature review process. Appendix B lists the 18 standards 
developed. Appendix C describes the proposed use of numeric data for 
selected quality indicators. A 59-^item list of references is 
included. (SLD) 
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INTRODUCTION 

Georgia's State I^egislature has mandated periodic evaluation of public 
schools and school systems throughout the state. As a part of this evaluation, 
schools are to be classified as "nonstandard, " "standard/' or "exemplary." 
The Center for Educational Research and Evaluation at the University of 
North Carolina at Greensboro has been contracted to propose standards, 
indicators of quality, data-collection procedures, and instruments for 
evaluation of Georgia's public schools. Because pertinent legislation requires 
that evaluation be comprehensive, in that it address all programs and 
services provided by Georgia's public schools, it has been necessary to develop 
standards, indicators, procedures, and instrumentation that could be used to 
evaluate a multiplidty of school functions and outcomes. 

This paper discusses the derivation of a set of standards and indicators 
for use in mandated school evaluation in Georgia. In developing the 
indicator system to be used in Georgia, initial consideration was given to 
assessing process indicators for each school program and service, in addition 
to indicators of school-wide educational outcomes and school characteristic**. 
Several factors contributed to early revision of this strategy, including 
considerations of the feasibility of reliable assessment within the legally- 
mandated constraints of the evaluation, considerations of cost, and of the 
likely intrusiveness of such assessments into the educational processes of the 
schools being evaluated. Due to these considerations, the initial 
concentration on program-by-program evaluation was modified, and 
formulation of standards and indicators for use in Georgia was focused on 
school-wide educational outcomes and the characteristics of effective schools. 
These two fod are conceptually dissimilar, in that educational outcomes deal 
only with evidence of the effects of education on students, while the 
characteristics of effective schools include both process factors and outcomes 
that involve persons other than students. The two fod are also supported by 
separate and distinct bodies of professional literature. However, the two fod 
are conceptually similar in that they both deal with school-wide factors that 
transcend programmatic boundaries. 

Organization of this Document 

Development of the standards and indicators for use in school 
evaluation in Georgia next proceeded in two phases. Rrst, a comprehensive 
list of standards and indicators associated with school quality was formulated 
through review of the research on school-wide educational inc^cator systems, 
and the research literahire on the characteristics of effective schools. We were 
purposefully indusive in developing this list of standards and indicators. 
Next, the initial comprehensive list of standards and indicators of school 
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quality was subjected to an extensive set of review and revision processes, to 
derive a set of standards and indicators for operational use. 

The next section of this pap^j describes the formulation of the initial, 
comprehensive list of standards and indicators of school quality. (A synopsis 
of the review of the professional literature from which these standards and 
indicators were derived is presented in Appendix A of this paper. The initial, 
comprehensive list of standards and indicators is presented in Appendix B of 
this paper.) The second section of this paper describes the revision and 
review processes used to derive the standards and indicators for operational 
use. The final section of this paper provides conclusions on major factors to 
be considered in translating the results of research into standards and 
indicators for operational use in school evaluation. 

DEVELOPMENT OF COMPREHENISVE LIST OF 
STANDARDS AND INDICATORS OF QUALITY FOR THE EVALUATION OF 
EDUCATIONAL OUTCOMES AND SCHOOL CHARACTERISTICS 

Assumptions 

Although the list of standards and indicators of school quality derived at this 
phase was intended to be inclusive, certain assumptions did underlie the 
development of the indicator system. The first and most basic assumption is that 
schools should be held resp'^nsible only for factors over which they have some 
degree of control. Thus, indicators dealing with factors such as funding were not 
included because schools could not modify such indicators at will. 

Other assumptions derive logically from the requirements of the enabling 
legislation. In particular, the legislation requires that evaluative information not 
available at the Department of Education be collected on-site in Georgia's schools and 
school systems by teams composed of education professionals and lay citizens. 
Teams must be composed of public school staff outside the local unit of 
administration that is being evaluated, lay citizens who are served by ti^at local unit 
of administration, and college and university faculty. Administrative and fiscal 
constraints dictate a limited range of subject-matter specializations among team 
members. Finally, standards, indicators, procedures, and instrumentatiort must be 
suffidentiy concrete and prescriptive that they can be used effectively by educators 
and citizens with a variety of specializations during a time-limited visit to schools 
and school systems. 

Another assumption underlying development of the initial indicator system is 
the belief that an evaluation should reflect the perspectives and interests of all 
stakeholders in the system. The primary purpose of a school is to achieve positive 
effects on students, thus the outcome indicator system developed at tiiis phase was 
designed to focus on outcomes for all groups of students. Likewise, indicators 
related to characteristics of effective schools focus on characteristics related to 
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effectiveness for all groups of students. Karmd (19S5), Felter (1989), and others have 
stressed the importance of assessing educational outcome indicators separately for 
minorities and females, since these groups have traditionally been less well served 
by public education systems. Many of the outcome indicators included in the 
original comprehensive list reflect special consideration for these groups. 

Tyler (1983) and others have noted that many educational outcomes are not 
normally distributed, thus the average or mean score may not be representative of 
the performance of a majority of students. Also, it is important to examine 
outcomes for all students, not just the "average" ones. For this reason, it was 
anticipated that the measures developed for some, if not all, of the outcome 
indicators would not be limited to average scores, but would also include the 
proportions above and below certain levels. 

The last assumption underlying the development of the initial indicator system 
concerns the manner in which benchmark criteria would be set for the outcome 
indicators ultimately chosen. The professional literature does not contain 
scientifically derived criteria for the level of attainment of the various outcome 
indicators that readily translate to categories labeled "standard or "exemplary." The 
literature generally includes an implicit or explicit assumpUon of "the more, the 
better." For example, graduation rate is a frequently dted outcome indicator, but the 
professional literature does not contain sdeniifically derived rules for what rate is 
"standard" or what rate is "exemplary." All such distinctions are based upon 
opinion. Clearly, a dropout rate of 0.0% would be wonderful, but, equally clearly, this 
is not a reasonable criteria of excellence to which schools should be held. Such 
criteria should be determined by stakeholders in the evaluation, and should be 
established in light of information concerning normative values for the state of 
Georgia and other states, and the criteria used by other stat^ and educational 
systems. (A process of policy capturing has l>een proposed for this purpose.) 

As stated earlier, the indicator system developed at this phase was intended to 
be inclusive and to reflect a comprehensive arr£iy of indicators. It was recognized 
that it might ultimately be infeasible to assess some of these indicators due to costs or 
logistic constraints. 



Development of Standards and Indicators of Educational Outcomes 

Educational evaluation has been referred to as "high stakes" evaluation by 
Popham (1975) and others. This designation refers to the fact that the intellectual, 
emotional, and physical well-being of hundreds or even thousands of learners 
can be influenced beneficially or adversely because of the results of an 
educational evaluation. The well-being of teachers and other educational 
professionals also depends upon evaluation, as does the peace of mind of the 
public It is, therefore, imperative that a comprehensive educatior^al evaluation 
be designed and conducted with the utmost rigor and conceptual clarity. The 
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ultimate goal of an education system is to effect positive change in learners, thus 
the measurement of student learning and development, as reflected in outcome 
Indicators, is essential for judging the effectiveness of schools and school 
systems. Since schools are responsible for promoting student learning and 
success in many different areas, multiple outcome indicators are needed to 
provide a comprehensive evaluation of schools* functioning (McMillan, 1989; 
Oakes, 1986). 

The system of indicators chosen for an evaluation is the backbone of the 
evaluation. Ideally, the Indicators will be chosen so that each measures a distinct 
aspect of the educational system, and taken together they provide information 
about how the systems' components function to produce overall results. In 
developing a system of educational indicators, it is essential that the variables 
chosen reflect the multiple purposes of educational systems (Glickman, 1987; 
McMillan, 1989; Oakes, 1986; Sirotnik, 1987). Educational evaluations conducted 
in the past have been criticized for exc^sive reliance upon standardized test 
scores as the sole indicator of student perfcrmance (cf.. Governor's Commission, 
1989; Glickman, 1987; McLean, 1989; Tyler, 1983). Professionals charge that this 
reliance results in a kind of "tunnel vision" which compromises the validity of 
the evaluation for even a limited range of educational outcomes, and leaves 
low-pHrforming schools little indication of how to effect improvement, except 
through drill on the spedfic variable(s) tested (cf., Cuban, 1983; Frechtling, 1989; 
Haertel, Haertel & Katzenmeyer, 1989; Worthen & Sanders, 1987). However, 
standardized test scores, when wisely used, provide useful, reliable, and objective 
informadon about student achievement. Tyler (1983) and others have noted that 
many educational outcomes are not normally distributed, thus the average or 
mean icore may not be representative of the performance of a majority of 
students. Also, it is important to examine outcomes for all students, not just the 
"average" ones. For this reason, it was anticipated that the measures developed 
for some, if not most, of the outcome indicators would not be limited to average 
scores, but would also include the proportions above and below certain levels. 

Haertel, Katzenmeyer, and Haertel (1989) stress that there is no single, 
accepted model for an educational outcome indicator system. The problem, 
then, is to formulate a comprehensive system of outcome indicators which v«ll 
validly reflect the multiple purposes of public education at the school level. The 
system developed at this phase was intended to reflect an inclusive array of 
indicators that were judged to be important and amenable to feasible assessment. 
Karmel (1985), Felter (1989), and others have stressed the importance of assessing 
educational outcome indicators separately for minorities and females, since these 
groups have traditionally been less well served by public education systems. In 
fact, court decisions resulting from litigation charging discriminatory practice, 
and policy positions design^ to protect the interests of minorities and females 
often require that outcomes be assessed separately for these groups (cf., Allen v. 
Alabama State Board of Education, 1985; Equal Employment Opportunity 
Commission, Civil Service Commission, Department of Labor & Department of 
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Jtistice, 1978; Gdden Rule Insurance Company v. Washbtum, 1984; Hardigan & 
Wigdor, 1989; McAllister, 1987). Many of the outcome indicate develop at 
this phase reflect special consideration for these groups. According to the Digest 
of EducatUmttl Smistics (National Center for Educational Statistics, 1989), the 
student population of Georgia's public schools is approximately 61% white, 38% 
black, <1% Hispanic, <1% Asian/Pacific Islander, and < 0.05% American Indian, 
Native Alaskan or Aleut. Although Asian students often perform very 
differentiy from other minorities on educational indicators, given the small 
nimibers of non-black minorities in Georgia's schools, it would be infeasible to 
assess outcomes separately for minorities other than blacks, and grouping aU 
non-whites together would be of questionable validity. For this reason, blacks 
are only minority for whom separate indicators were proposed at this phase of 
development. 

Recommendations for outcome assessment in the professional literature 
and the practice of outcome assessment in most educational regulatory systems 
typically focus on outcome indicators rather than outcome standards. However, 
the types of indicators recommended or chosen can be used inductively to derive 
implicit standards. In developing a system of outcome indicators for the 
Educational Performance Recognition Program in Virginia, McMillan (1989) 
identified seven broad goals of public education in Virginia: 1) improving 
school completion rate, 2) preparing students for post-secondary education, 3) 
preparing students for work, 4) increasing special education student's living 
skills and opporttmities, 5) educating elementary school students, 6) educating 
middle school students, and 7) educating secondary school students. These 
seven goals reflect the multiple academic purposes of public education generinly 
cited in the professional literamre, and reflect what Herron (1980) refers to as 
"school skills." However, education has goals for students beyond the purely 
academic. The professional literature abounds with calls for assessment of such 
"non-academic" outcomes as students' valuing of the environment, or valuing 
of cultural diversity (cf., Bowen, 1980; Hargreaves et al., 1989; Herron, 1980; NCA 
Conunission on Schools, 1985; Pace, 1984). Georgia's Quality Basic Education Act 
(Georgia School Laws, 1987, pp. 46-47) also reflects several such goals, including 
"ensuring that each student is provided ample opportunities to develop 
competencies necessary for lifelong learning, ...to participate actively in the 
governing process, to protect the environment,., and to be an effective worker 
and responsible citizen." Herron (1980) refers to ti\ese as "life skills," stating that 
they are aimed at preparing students for the multiple life roles which they are 
likely to encounter in our society. When this category is added to McMillan's 
(1989) list, ti\e resulting outcome goals for students can be induced from 
indicators recommended or used in the professional literature. 



A Researdt-Based Attribute Stnicttire for School Accountability 



7 



Thus, the outcome goals of education which served to organize the development 
of standards and indicators of educational outcomes in this project were: 

1) Improving student's school completion rate. 

2) Preparing students for post-secondary school life: 

A. Preparing students for continued schooling. 

B. Preparing students for work. 

C. Preparing students for multiple life roles. 

3) Providing students with school experiences appropriate to their ages, 

developmental levels and skill levels: 

A. Educating elementary school students. 

B. Educating middle school students. 

C. Educating secondary school students. 

D. Increasing special education students' living skills and 

opportunities. 

Standards and indicators for each of these outcome goals were developed 
through review of the pertinent professional literature. To accomplish the 
comprehensive review of the relevant professional literature upon which the 
comprehensive outcome indicator system was based, a computeiized literature 
search (ERIC) was conducted. The sources identified in the literature search were 
augmented by a review of the educational evaluation practices of other states 
and of professional papers presented at the latest meetings of the American 
Educational Research Association and the National Council on Measurement in 
Education. A synopsis of liie results of review of the professional literature for 
each of the outcome goals of education listed above appears in Appendix A of 
this paper. An integrated summary list of standards and indicators for 
evaluation of educational outcomes and school characteristics appears in 
Appendix B of this paper. 



Development of Standards and Indicators of School Characteristics 

Research grounded in the "effective schools" paradigm has resulted in 
what Haertel, Katzenmeyer, and Haertel (1989, p. 17) refer to as a "loose 
consensus on the characteristics of effective schools." These characteristics are 
not outcomes per se; rather they are features of schools which cross subject area 
lx>undaries and which have been found to be associated with positive student 
outcomes, particularly achievement outcomes in math and reading. Ralph and 
Fennessey (1983, p. 694) describe the results of most studies on the characteristics 
of effective schools as including "some combination of: 1) strong administrative 
leadership, 2) a safe and orderiy school climate, 3) an emphasis on basic academic 
skills, 4) high teacher expectations for all students, and 5) a system for 
monitoring and assessing pupil performance." This is widely referred to as the 
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"ave-factor model." The effective schools perspective has been criticized as being 
more a rhetoric of reform than a scientific evaluation modd (Haertel et al., 1989; 
Ralph and Fennessey, 1983). The reliability and validity of the measures used to 
classify schools as more or less effective have been questioned, and the lists of 
characteristics of effective schools used by different investigators have been 
criticized as inconsistentiy defined and pooriy justified empirically (Cuban, 1983; 
Haertel et al., 1989; Purkey and Smith, 1983; Ralph and Hennessey, 1983). 
Wimpelberg, Teddlie, and Stringfield (1989) note ti\e restricted selection of sites 
for research on school effectiveness. Many of tiie studies of so-called "effective 
schools" are based on detailed case studies of schools which have been identified 
as effective in producing outcomes, in tiie absence of corresponding study of 
ineffective schools; thus, tiie factors identified as characteristic of effective 
schools may also be characteristic of many ineffective schools (Gottfredson, 1985; 
Hallinger and Murphy, 1985; Lewis, 1986; Wynne, 1984). 

Several authors agree that the implications for action derived from the 
effective schools research paradigm are undear (Cuban, 1983; Haertel et al., 1989; 
Pugh, 1989). As Haertel et al. (1989, p. 18) state: "Ineffectual principals cannot 
become strong leaders by a simple act of will, nor can teachers change their 
expectations overnight. School climate is ...resistant to change by administrative 
fiat." Even if the distinctive features of effective schools could l» determined 
imambiguously, it does not necessarily follow that other schools could become 
more effective by emulating them. After administration of an instrument which 
was carefully developed to measure the degree of "effective school" 
characteristics present in a school to a state-wide sample of elementary schools, 
Maruyama, Deno, Cohen, and Espin (1989) found most of the characteristics to 
have insignificant correlations with reading achievement, and foimd no 
correlations greater than 0.27. Despite tiiese criticisms, "effective schools" 
researchers state that the research literature has repeatedly failed to show any 
consistent relationships between variations in curriculum, instructional 
practices, or facilities and student achievement; practices that seem to work in 
one school do not work in another. They state that U\is is because characteristics 
of the learning environment and teachers have a stronger mfluence on student 
outcomes tiiaii do specifics of the facilities, curriculum, or instructional practices 
used (Austin, 1979). 

It is not possible to delineate cause and effect definitively in educational 
systems: schools are "loosely coupled systems" (Meyer & Rowan, 1978; Tyler, 
1983). The aitidsms above are adequate reason to question tiie wisdom of 
including only "effective schools" factors in an educational evaluation system, 
but do not predude their utility as a part of a comprehensive evaluation system 
based upon multiple indicators. 

Purkey and Smith (1983) completed a critical review of the effective 
schools literature and derived a set of nine organizational-structure variables 
and four process variables related to effective schools. This set of variables has 
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been described as a more coherent model than the list of five factors typically 
dted in effecdve sdiools studies and is considered more appropriate for a 
comprehensive school evaluation (Haertel et al., 1989). The nine organizational- 
structure variables identified include: 1) autonomy of school-site management, 
2) instructional leadership, 3) staff stability, 4) curriculum articulation and 
organization, 5) schoolwide staff development, 6) parental involvement and 
support, 7) sdioolwide recognition of academic success, 8) maximized learning 
time, and 9) district support. The four process variables are: 1) collaborative 
plaxming and collegial relationships, 2) sense of community, 3) dear goals and 
high expectations, and 4) order and discipline. Several of these variables are 
already assessed through legal compliance monitoring in Georgia. The 
articulation and organization of the curriculum, (Purkey and Smith, 1983) are 
addressed by a mandated curriculum in Georgia. Evaluation of schoolwide staff 
development is similarly mandated. The variable Purkey and Smith identify as 
district support indudes consideration of the level and types of fincndal and 
other support and guidance provided to the school by the district office. 
Although these factors undoubtedly affect shident outcomes at a school, they are 
not within tiie control of the schools. This renders them inappropriate criteria 
upon which to evaluate the schools (Haertel et al., 1989), thus they wiU not be 
developed as educational indicators in this chapter. Excluding the variables 
considered inappropriate for use in Georgia, the factors dted and described by 
Purkey and Smith (1983) and by Purkey and Degen (1985) encompass most of the 
findings of current studies on effective schools. 

These factors are: 

1. Autonomy of school-site management 

2. Instructional leadership 
3.Staff stabiUty 

4. Parental involvement and support 

5. Schoolwide recognition of academic success 

6. Maximized learning time 

7. Collaborative planning and collegial relationships 

8. Sense of community 

9. Clear goals and high expectations 

10. Order and disdpline 

Standards and indicators for each of these tert factors were developed 
through review of the pertinent professional literature. To accomplish the 
comprehensive review of the relevant professional literature upon which the 
comprehensive school characteristics indicator system was based, a computerized 
literature search (ERIC) was conducted. The sources identified in the literature 
search were augmented by a review of the educational evaluation practices of 
otiier states and of professional papers presented at the latest meetings of the 
American Educational Research Association and the National Council on 
Measurement in Education. A synopsis of tiw results of review of the 
professional literature on each of the factors characteristic of effective schools 
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listed above appears in Appencdx A <rf this paper. An integrated summary list of 
standards and indicators for evaliiation of educational outcomes and school 
characteristics appears in Appendix B of this paper. 

As stated earlier, the educational outcome and school characteristics 
indicator system developed at this phase was intended to be inclusive and to 
reflect a comprehensive array of indicators. It was recognized that it might 
ultimately be infeasible to assess some of these indicators due to costs or logistic 
constraints. 



REVISION OF STANDARDS AND INDICATORS 
OF QUAUTY FOR THE EVALUATION OF 
EDUCATIONAL OUTCOMES AND SCHOOL CHARACTERISTICS 

As described above, the first phase of the development of standards and 
indicators for use in school evaluation in Georgia involved compilation of a 
comprehensive list of educational outcomes and school characteristics. During the 
second phase of development, this comprehensive list was subjected to an extensive 
review and revision process, which is described in this section of the paper. 

Assumptions 

In addition to the assumptions upon which formulation of the initial 
comprehensive list of standards and indicators was based, some additional 
assumption formed a basis for the revision of standards and indicators for operation 
use. 

Sirotnik (1987) and others have recognized the effect of "respondent burden" on 
the quality of information gathered in an educational evaluation. Increasing the 
volume, frequency, or duration of data collection beyond a certain limit produces an 
unfavorable impact on the reliability and validity of the information gathered, due 
to missing data, respondent fatigue and loss of attention, and impairment of the 
cooperative attitude of respondents. Thus, indicators and measures must be selected 
to yield the maximum amount of reliable and essential data with the least burden 
upon respondents at the school level, and with the least disruption of the regular 
school routine. 

Derivation of the revised set of standards and indicators of quality for 
evaluation of educational outcomes and school characteristics has also been 
guided by a set of assumptions that derive logically from the requirements of 
valid and reliable measurement, coupled with feasibility. Specifically, since the 
student populations served by any single school program are finite and relatively 
small, estimation of the proportion of students who have a specific opinion, 
belief, abiUty, or understanding with any acceptable degree of predsion would 
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require collection of data from virtually every student served by the program. 
Interviews with only a few randomly selected students caimot validly represent 
the views of all students. Considerations of cost and time render census 
interviews or written surveys of all students in a school infeasible; thus 
standards and indicators which require such student data were eliminated at this 
phase of development, although such standards and indicaiors were often 
considered important by all parties providing feedback on the initial 
comprehensive of standards and indicators. A parallel argument rendered 
surveys of parents or members of the community infeasible. 

A final assumption concerns the timing of evaluation activities in Georgia 
schools. Once the Georgia Comprehensive Evaluation System (CES) is fully 
developed and in place, schools will be visited and reviewed once every five years. 
An assumption underlying developntent of the indicator system at this phase is that, 
when the CES is fully operational, the evaluation review will consider the status of 
many indicators across the years since the last site visit, rather than limiting 
consideration to the outcomes of a single year* . This would protect schools from 
receiving an unduly harsh evaluation because they happen to be reviewed during a 
year when some outcome is unusually low for them. Considering outcomes across 
all years since the last site review would also help schools keep an ongoing focus on 
striving for quality of outcomes and will maximize measurement reliability. During 
the developmental years of the CES, schools will not have maintained five-year 
longitudinal records on all relevant student outcomes. Altiiough schools could 
possibly recoi\struct historical records of some, if not all, outcome indicators, this 
would likely impose an undue hardship upon the schools' administrators and 
record keeping staff. It was thus anticipated tiiat, during the developmental phases 
of the CES, evaluations would focus on data from the year(s) since implementation 
of the Quality Assessment Module of the CES, and data from otiier years as available. 



Methodology 

During the this phase of development, the initial comprehensive list of 
standards and indicators was subjected to extensive review processes. These 
processes involved repeated iterations incorporating feedback from stakeholders, as 
well as considerations of validity and reliability of measurement, constraints of the 
legislative mandate, and considerations of data availability and accessibility, costs, 
and respondents' preferences for modes of data collection. To accomplish review on 
these diverse considerations, several distinct review panels were composed: 

1. A Technical Advisory Board (TAB) was composed of five nationally 
prominent experts in educational research methodology and evaluation- This 
panel convened two to three times a year with the project's professional staff, 
and conferred frequentiy by mail and telephone, to provide critical feedback on 
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methods for soUdting feedback from relevant stakeholders in the evaluation, 
alternative methods for measuring proposed indicators of quality, and the 
balance and adequacy of the indicator system being developed. 

2. A Departmental Technical Advisory Board (DTAB) was composed of 
professionals from the Georgia Department of Education and Georgia State 
University. Department of Education personnel were chosen to represent 
several areas within the Department, including the Division of Standards and 
Evaluation, the Division of Assessment, the Divi&ion of Strategic Planning, 
and the Office of Instructional Programs. This group convened approximately 
twice a month to review the development of the piojecv and to provide 
feedback on consistency of products wiJi the legislative mandate, 
considerations of data availabilit}' and accessibility, and likely costs of 
measurement of proposed indicators. The DTAB also provided essential 
information on the political context of the evaluation, and any redundancy of 
proposed standards and indicators with other regulatory or evaluative systems 
in the state. 

3. A Comprehensive Evaluation System Advisory Committee (Advisory 
Committee) was composed of forty Georgia educators and citizens. Tb:s group 
convened t%"'o to three times a year to provide feedback on the availability and 
accessibility of data, respondent's preferences for modes of data collection, the 
clarity and meaning of terminology used in standards and indicators to 
educators and citizens in Georgia, and the political context of the evaluation in 
local school systems. School systems represented on Advisory Committee also 
served as pilot- and field-test schools as instruments and procedures for 
assessing proposed standards of quality were developed. 

4. A Steering Committee of the Comprehensive Evaluation System Advisory 
Committee (Steering Committee) was composed of fifteen members of the 
Advisory Committee. The Steering Committee convened approximately 
monthly, and a member of the Steering Committee served on the DTAB 
(described above). The Steering Committee provided feedback on the 
availability and accessibility of data, respondent's preferences for oodcs of data 
collection, the clarity and meaning of terminology used in standards and 
indicators to educators and citizens in Georgia, and the political context of the 
evaluation in local school systems. The Steering Committee worked closely 
with the project's professional staff and the DTAB to ensure that the standards, 
indicators, and measurement procedures ultimately developed were 
acceptable to Georgia educators and citizens. 

Each of the groups described above reviewed and reacted to the initial 
comprehensive list of standards and indicators of educational outcomes and school 
characteristics. This led to extensive revision of the standards and indicators. As 
discussed above under the assumptions of this phase of the project, it was judged 
ipfeasible to reliably measure indicators which required collection of survey or 
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interview data from students or parents. These indicators were reluctantly 
abandoned, despite the fact that they were considered important by all reviewers. 
Deletion of indicators requiring such data from students also led to the deletion of 
one entire standard. 

Feedback from the DTAB and the Steering Committee led to the rewording of 
many of the standards and indicators. In some instances, this rewording was 
necessary to make an indicator applicable to Georgia. For example, Georgia awards a 
College Preparatory Seal of Endorsement, rather than an Honors Diploma, to 
students completing a rigorous course of study; thus it was necessary to reword 
indicators which had initially specified "Honors Diploma/' Other wording changes 
were necessary to make indicators dear and acceptable to educators and the public in 
Georgia. For example, the phrase "is acceptable" (as in "The graduation rate is 
acceptable") did not seem incon^stent with sound measurement practice to the 
project's professional staff or to the TAB, but the Steering Committee strongly felt 
that the phrase implied that subjective, individualistic judgements would be made 
on the indicators of quality. For this reason, the wording of the phrase was changed 
to "meets or exceeds the established level" for most indicators. Similar wording 
changes were made in many of the standards and indicators of quality. 

Although the project's professiona? staff and the TAB supported assessment of 
certain indicators separately for minorities, females, and special education students, 
stakeholder groups foimd such indicators unacceptable for several reasons. First, the 
evaluation model being developed called for indicators of school-wide educational 
outcomes and school characteristics, and these groups felt that such indicators were 
too limited in scope, and of limited importance. Second, such indicators were judged 
incompatible with the political context of the evaluation Third, it was judged that 
measurement of such indicators would substantially increase the respondent burden 
and cost of data collection, since Georgia schools and the Department of Education do 
not routinely maintain student data distinguished by race or gender. These 
considerations led to fhe deletion of all indicators which addressed outcomes for 
minorities, females, or special education students. 

A number of other types of indicators were eliminated due to being judged of 
limited importance, and/or infeasible to assess due to considerations of cost and 
respondent burden. These included indicators involving course enrollment rates, 
post-secondary activities of students, and feedback from schools to which stuaents 
matriculated. Indicators involving students' participation in extracurricxilar 
activities, mati\ and science fairs, and the arts were judged by the project's 
professiona! staff and the TAB to be of questionable validity, since factors besides 
school quality (such as urban versus rural locations schools) ntight contribute 
substantially to variance in mese indicators. 

Synthesis of the feedback from each of the groups reviewing tiie initiai 
comprehensive list of standards and indicators led to the formulation of a 
substantially reduced list of revised standards and indicators of educational outcomes 
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and school characteristics. As the next phase in the development of the projects data 
sources and data elements were proposed for each indicator of quality on the revised 
list. These revised standards and indicators, with proposed data sources and data 
elements, were then reviewed by each of the groups listed above. This review led to 
further minor revision of some indicators, and to revision of some proposed data 
elements and sources. 

At this phase in the development of the project, three major data sources for 
assessing indicators of quality had been identified: 1) data available within the 
Georgia Department of Education (such as standardized test scores and graduation 
rates), 2) data which could be gathered from schools through a mailed census 
questionnaire (such as attendance rales, and rates of suspension and expulsion), and 
data which would be collected through the mandated on-site visits by the evaluation 
teams (including data from interviev^ and written surveys of school personnel, data 
collected by team members through observations and review of documentary 
evidence.) To further define the data soimres and data elements for many of the 
indicators of quality, a series of visits to selected schools and systems in Georgia were 
conducted. At each school, teachers and other school personnel were interviewed 
regarding the clarity of the wording of indicators which required data on the 
opinions or experiences of school personnel. They were also asked about any 
concerns they would have (e.g., confidentiality) in providing information in 
response to these indicators, and their preferences for modes of data collection (e.g., 
written survey or interview). Schoc4 and system personnel were interviewed 
regarding the availability and accessibility of certain types of nimieric data, such as 
data on rates of staff turnover. After analyses of the results of these school visits, 
indicators and associated data sources and data elements were again reviewed in 
light of considerations of respondent burden and respondent's preferred modes of 
data collection. 

The processes of review by each of the groups described above, and subsequent 
revision, have been followed through each step of the process, including the 
development and pilot testing of the mailed census questionnaire, and the 
development of instruments and procedures for school site-visits (which are to be 
pilot tested this spring). Although the process has been time-consuming and often 
required the resolution of differences between conflicting values and judgemer^s, we 
feel it has allowed development of an evaluation system with technical integrity that 
will ultimately prove acceptable to all relevant stakeholders in the evaluation. 
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SUMMARY 

This paper has described the effort of translating the results of effective schools 
research, the recommendations of professional councils and bodies regarding school 
evaluation, and the results of research on educational outcomes, into a system of 
standards and indicators of educational outcomes and school characteristics for 
operational use in a state-mandated school evaluation system. This translation 
process involved repeated iterations incorporating feedback from stakeholders, as 
well as coi^iderations of validity and reliab*Uty of measurement, constraints of the 
legislative mandate, and considerations of data availability and accessibility, cost, and 
respondents' preferences for modes of data collection; in an effort to develop an 
operational system with technical integrity which met the requirements of the law 
and was acceptable to stakeholders in the evaluation. 

From our experiences one may draw several conclusions regarding the major 
factors which must be considered in developing such a system: 

1. Ensure adequate opportunities for obtaining feedback from all relevant 
stakeholders in the evaluation. Although stakeholder participation is discussed in 
every elen^entary evaluation textbook, it cannot be over-emphasized. Stakeholders 
must have opportunities to participate in each step of the development process. 
Even though systems like the one we are developing may be legally mandated, their 
survival and effectiveness depend critically on stakeholder acceptance. Our 
experience indicates that stakeholder participation is more constructive and 
productive if it is relatively structured: if staiceholders clearly understand their role 
in the development of the evaluation system, and know in advance when they will 
be provided opportunities to provide f^back, what they will be reviewing at each 
such opportunity, how they arc expected to conduct their reviews or otherwise reach 
conclusions or make recommendations, and what sorts of factors they are expected to 
consider in conducting reviews, making recommendations, and reaching 
conclusions. 

2. Ensure adequate consideration of the political context of the evaluation. This 
factor is, of course, inextricably linked with stakeholder participation, and relates 
similarly to the viability of the evaluation system ultimately developed. 

3. Take careful stock of the "respondent burden" which will be imposed by the 
evaluation system. Sirotrdk (1987) and other s have recognized the effect of 
"respondent burden** on the quality of information gathered in an educational 
evaluation. Gncreasing the volume, frequency, or duration of data collection beyond 
a certain limit produces an unfavorable impact on the reliability and validity of the 
information gathered, due to missing data, respondent fatigue and loss of attention, 
and impairment of the cooperative attitude of respondents.) The stakeholders in the 
evaluation system we are developing also repeatedly remind us of this factor. This is 
a factor which must be considered at each step in developing an evaluation system. 
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4. To ti\e greatest extent possible, avoid develoving an evaluation system which vrill 
inevitably disrupt the entity or process it is intended to evaluate. Stakeholders in the 
evaluation system we are developing also repeated remind us to develop a system 
which is minimally intrusive into the school's task of educating students. Indicators 
and measures must be selected to yield the maximum amount of reliable and 
essential data with the least burden upon respondents at the school level, and with 
the least disruption of the regular school routine. 

5. Give ongoing considerations to the constraints on the evaluation, including 
constraints related to cost, personnel, the timeline for development, and the 
timeframe for implementation. 

6. Ensure ongoing assessment of the reliability and validity of the evaluation 
procedures being developed. Often, conflicting opinions over priorities, values, and 
judgments arise during development of an evaluation system. Such differences 
must be reconciled in a manner which does not impair the integrity of the 
evaluation system. Not only must the reliability and validity of each indicator be 
monitored, ti\e balance, adequacy, and integrity of the overally system must be 
continually examined. 
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APPENDIX A 

REVIEW OF LITERATURE ON STANDARDS AND INDICATORS OF 
QUALITY FOR THE EVALUATION OF 
EDUCATIONAL OUTCOMES AND SCHOOL CHARACTERISTICS 

REVIEW OF LITERATURE ON STANDARDS AND INDICATORS OF 
QUALITY FOR EDUCATIONAL OUTCOMES 

As discussed in the body of this paper, the outcome goals of education 
which served to organize the development of standards and indicators of 
educational outcomes for this project were: 

1) Improving student's school completion rate. 

2) Preparing students for post-secondary school life: 

A. Preparing students for continued schooling. 

B. Preparing students for work. 

C. Preparing students for multiple life roles. 

3) I*roviding students with school experiences appropriate to their ages, 

developmental levels and skill levels: 

A. Educating elementary school students. 

B. Educating middle school students. 

C. Educating secondary school students. 

D. Increasing special education students' living skills and 
opportunities. 

Standards and indicators for each of these outcome goals were developed 
through review of the pertinent professional literature. To accomplish the 
comprehensive review of the relevant professional literature upon which the 
comprehensive outcome indicator system was based, a computerized literature 
search (ERIC) was conducted. The sources idaitified in the literatiire search were 
augmented by a review of the educational evaluation practices of other states 
and of professional papers presented at the latest meetings of the American 
Educational Research Association and the National Council on Measurement in 
Education. A synopsis of the results of review of the professional literature on 
each of the outcome goals of education appears below. An integrated summary 
list of standards and indicators for evaluation of educational outcomes and 
school characteristics appears in Appendix B of this paper. 

Improving School Completion Rate 

Georgia's conunitment to improving the graduation rate of its students is 
evidenced by the fact that efforts "to improve school completion" were a priority 
focus of the GDE State Strategic Planning Team for 1989 (Rogers, March 1989). 
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Dropout rates and graduation rates are frequently assessed in state, federal, and local 
outcome Indicator systems (c.f. Blank, 1989; Califomia Department of Education 
1989a, 1989b, 1989c; Oakes, 1986; OERI, 1988) or recommended in the professional 
literature (c.f. Glickman, 1987; Governor's Commission, 1989; NCA Commission on 
Schools, 1985). Both graduation rate and dropout rate address students* completion 
of schooling. Students who drop out clearly cannot complete their schooling. 
Graduation rate, however, is influenced by the dropout rate, the rate of student 
transfers, and retention rate. Retention may benefit a student's education, transfer 
may not be detrimental, but dropping out is certainly detrimental. For this reason, 
both the dropout rate and the graduation rate are used here as Indicators. Minority 
students are often at special risk of failure to complete their education, thus both 
overall dropout and graduation rates and minority dropout and graduation rates 
were included as indicators. The state of Virginia, in their Standards of Quality for 
Public Schools, considers both overall dropout rate and minority dropout rate as 
indicators directed toward improving school completion rate (McMillan, 1989). The 
Council of Chief State School Officers also includes graduation rates and dropout 
rales, for all students and for minorities, in its recommended system of educational 
outcome indicators (Blank, 1989). 

Absenteeism is also related to failure to complete school, and is a frequently 
assessed indicator. However, absenteeism is also a barrier to the broader task of 
educating students, whether they complete their course of study or not. For this 
reason, absenteeism is addressed below, in the sections dealing with educating 
students at all levels, rather than in this section. 

Student performance on competency tests is also a frequently assessed outcome 
indicator (Haerlel, Katzenmeyer, & Haerld, 1989). Virginia uses the two-year pass rate 
(percent of students who pass the test within two years) on a basic competence test 
required for graduation in Virginia, as an indicator. The Maryland Governor's 
Commission on School Performance (1989) has also recommended that the 
percentage of students who pass tenth-grade competency tests be assessed as an 
outcome in that state's comprehensive educational evaluation system. Over 20 other 
states collect data on student competency test scores (OERI, 1988). The state of Georgia 
requires that students achieve a passing score on the Basic Skills Test, initially given 
in the fall of the tenth grade, in order to graduate. Students may repeat the test if 
necessary to achieve a passing score. The initial pass rate on the Basic Skills Test was 
included as an indicator here because it will be simpler to assess and more 
straightforward to interpret. As minorities typically have lower scores on 
standardized tests, the minority pass rate on the Basic Skills Test is also included as an 
indicator. Initial pass rates on the Basic Skills Tests were included in this section 
because of the likelihood that students who initially fail the test will become 
discouraged and are at inaeased risk of dropping out of school. 

Low achievmg students are at special risk of dropping out, and Tyler (1983) and 
others have stressed the need to assess the proportion of students who are low 
achievers, rather than just average scores, when considering • tudent outcomes. 
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Vir^nia considers the percent of students scoring in the lowest quartile of the ITBS 
(Iowa Tests of Basic Skills) or TAP (Tests of Achievement and Pro6dency) 
(depending on grade) as an indicator related to school completion. CSeorgia also uses 
the rreS and TAP, and these indicators were Included. Because minorities typically 
have lower scores on standardized tests, the percent of minorities scoring in the 
lowest quartile of the ITBS or TAP (depending on grade) is also included as an 
indicator. 



Preparing Students for Post-Secondaxy School Life 

Preparing Students for Continued Schooling 

Preparation for college is facilitated by completion of a particularly rigorous 
course of study in high school. The Governor's Commission on School 
Performance (1989) has recommended use of the percentage of students who 
complete a rigorous high school program, as indicated by the Advanced Studies or 
Honors diploma, as an outcome Indicator in Maryland's comprehensive 
educational evaluation system. The state of Virginia also uses the percent of high 
school graduates receiving the Advanced Studies Diploma as an indicator to be used 
in assessing whether a s»vhool is adequately preparing students for college 
(McMillan, 1989). The percent of students, and of minorities, receiving an Honors 
Diploma was included as an indicator of whether a secondary school is adequately 
preparing students for college. 

Scores on the Scholastic Aptitude Test (SAT) or the American College Test 
(ACT) are also frequently used as indicators (c.f. Haertel, Haertel & Katzenmeyer, 
1989; OERI State Accountability Study Group, 1988). The state of Vir^nia uses the 
percent of 11th and 12th graders taking the SAT, and the percent of llth and 12th 
grade minorities taking the SAT, as indicators to be used in assessing whether a 
school is adequately preparing students for college (McMillan, 1989). The percent of 
all students, and minorities, taking the SAT or ACT, as well as the average scores of 
all students and minorities on these tests, were included here as outcome indicators 
for sacondary schools. 

The Office of Educational Research and Improvement (OERI) (1988) 
recommends collecting data on the percentage of students who meet the state's 
university entrance requirements as an educational outcome indicator. This was 
included as a secondary school indicator, both for all students and for minorities. 
The proportion of graduating secondary school students who qualify for 
scholarships and awards also presents evidence of how well a secondary school has 
prepared students for college, and was included as an indicator. 

Patterns of course enrollment provide evidence of the adequacy of both 
secondary schools* and middle schools' preparation of their students for college. 
Rates of enrollment in advanced math, science and foreign language courses are 
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frequently assessed outcome indicators, and dearly relate to preparation for college 
(c£. Haertel Haertel & Katzenmeya:, 1989; OERI State AccountabiUty Study Group, 
1988). The state of Virginia uses the following indicators in assessing whether a 
school is adequately preparing students for college: the percent of 8th graders taking 
a foreign language, the percent of 8th graders taking algebra, and the percent of 
secondary school students taking one or more advanced placement or college level 
courses (McMiUan, 1989). The CouncU of Chief State School Officers includes 
advanced course exurollment rates, for all students and for minorities, in its 
recommended system of educational outcome inr'iotors (Blank, 1989). The 
California Department of EducaHon (1989a, 1989b, 1989c) and the Greensboro Public 
School System (1989) include assessment of enrollment patterns by race and gender 
in advanced courses in their system of quality indicators. Since foreign language and 
algebra are typically offered in middle schools, rates of enrollment in these courses 
were included as indicators of the degree to which middle schools are preparing 
students for college. Enrollment rates in advanced placement courses, foreign 
language courses, and advanced science and mathematics courses were included as 
indicators for secondary schools. Females are historically underrepresented in 
science and mathematics courses, thus rates are considered separately for females as 
well as minorities for these courses. 

Standardized norm- and criterion-referenced achievement test lesuUs are the 
most widely used student outcome indicator, used in virtually every state, federal 
and local educational indicator system . Above-average achievement is a necessary 
foundation for college, and is an indicator available at every grade level. The state of 
Virginia uses the percent of students scoring in the upper quartile of the ITBS or the 
TAP (depending on grade) as an indicator to assess whether a school is adequately 
preparing students for college (McMiUan, 1989). Georgia also uses the ITBS and TAP. 
The proportion of all students, and minorities, scoring in the upper quartile of these 
norm referenced tests was included as indicators. 

The Governor's Commission on School Performance (1989) has also 
recommended use of the post-secondary attainment of graduates of Maryland high 
schools as an outcome indicator in Maryland's comprehensive educational 
evaluation system. Bowen (1980) and the NCA Commission on Schools (1985) have 
also stressed the need to include assessment of outcomes concerning post-graduation 
sUtus of school alumni. The North Carolina State Board of Education (1989) 
includes assessment of post-graduation status in its educational indicator system, and 
over a third of the states collect some data on post-graduation status (OERI, 1988). 
The state of Virginia includes scrutiny of the percent of graduates who attend a two- 
or four-year college the year subsequent to graduation in assessing whether a school 
is adequately preparing students for college (McMillan, 1989). The Council of Chief 
State School Officers includes post-secondary school status, for all students and for 
minorities, in its recomniended system of educational outcome indicators (Blank, 
1989). The percent of graduates, and minority graduates, who subsequently attend 
college were included as outcome indicators. 
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Preparing Students for Work 

Although preparing students for work is an important and widely 
recognized function of schools, outcome indicators relating spedfically to this 
function are cmly beginning to be addressed in the professional literature. The 
state of Virginia uses the following indicators to assess the degree to v/hich 
schools adequately prepare students for work: the percent of vocational 
education students completing tfieir vocational education program, the percent 
of minority vocational education students completing their vocational education 
program, the graduation rate of vocational education students, the percent of 
12th-graders who have taken a vocational aptitude test or interest inventory at 
some time, and the percent of graduating students who do not attend post- 
secondary education who are employed after graduation (McMillan, 1989). The 
graduation rate of minority vocational education students was Included as an 
indicator, at the request of members of the CES State Advisory Board. 

Both tiie NorUi Carolina State Board of Education (1989) and tiie 
Governor's Commission on School Performance (1989) have recommended data 
on the post-secondary activities of graduates of the states' secondary schools as 
outcomes in their comprehensive educational evaluation systems. Bowen (1980) 
and the NCA Commission on Schools (1985) stressed the need to include 
assessment of outcomes concerning post-graduation status of school alumni, of 
students, and The Council of Chief State School CXficers included assessment of 
post secondary school status, for all students and for minorities, in its 
recommended system of educational outcome indicators (Blank, 1989). 
Indicators related to rates of post-graduation employment should, of course, be 
adjusted for local labor market conditions, as reported by local state employment 
service offices. The percent of students compIetL.g vocational education 
programs was included as an indicator for high schools. 

Preparing Students for Multiple Life Roles 

Britell (1980) stresses (tie importance of assessing whether students have 
acquired tiie skills required in order to function as citizens in the society in 
which titey live when evaluating educational outcomes. In order to function in a 
technologically advanced society committed to a democratic government, 
students must examine alternatives and make decisions. Similarly, Herron 
(1980) stresses the importance of assessing whether students are being adequately 
prepared to cope witii the various life roles witii which they will likely be faced- 
the roles of learner, producer, individual, citizen, consumer, and family 
member. Herron refers to these as "life skills," and states that tiiey requii^ more 
than (tie mere acquisition of knowledge. They also transcend subject area 
boundaries. The state of Georgia has a strong commitment to student outcome 
goals which reflect a focus on tiie multiple life roles of citizens in this society, as 
demonstrated in its adopted Quality Basic Education Act (Georgia School Laws, 
1987, pp. 46-47). The Quality Basic Education Act includes tiie following "life 
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skills'* goals: The student will be provided ample opportunity to develop the 
competencies necessary to: 1) recognize the need for lifelong learning, 2) 
participate actively in the governing process and community activities, 3) protect 
the environment and conserve public and private resources, 4) seek to maintain 
sound physical and mental health, and 5) be an effective worker and responsible 
dtizen. These goals clearly reflect a focus on preparing students to function in the 
roles of learner, producer, individual, citizen, consumer, and family member, as 
delineated by Herron (1980). Bowen (1980) has called for assessment of students' 
aesthetic sensibility, practical competence in citizenship and family life, and their 
emotional and moral development, when evaluating the outcomes of 
educational programs. Similarly, Pace (1984) has called for assessment of 
students' appreciation of the arts, citizenship, responsibility, and tolerance of and 
appreciation of other cultures. The state of Virginia has demonstrated a 
commitment to such goals, and is considering including assessment of outcome 
indicators of student's self-concept, attitudes toward school and learning, 
citizenship, valuing of cultural diversity, and knowledge and appreciation of the 
arts, in its comprehensive evaluation system. Maryland's Governor's 
Commission on School Performance (1989) and the NCA Commission on 
Schools (1985) have advocated the inclusion of indicators of students' 
appreciation of the arts, appreciation of cultural diversity, sense of responsibility 
to others, self concepts, attitudes toward learning, and understanding of the 
workings of the country's political system in comprehensive educational 
evaluation systems. The Virginia system also includes assessment of students' 
physical fitness (McMillan, 1989) as does the Dade County, Florida school 
recognition program (Haertel, Katzenmeyer, and Haertel, 1989) and the 
California system (1989a, 1989b, 1989c). Although the educational outcomes 
listed above were universally considered important by all reviewers, they are 
very difficult to measure reliably and validly, and would require extensive data 
collection from students. 

PROVIDING STUDENTS WITH SCHOOL EXPERIENCES APPROPRIATE TO 
THEIR AGES, DEVELOPMENTAL LEVELS, AND SKILL LEVELS 

Educating Elementary, Middle, and Secondary Students 

Standardized norm- and criterion-referenced achievement test results are 
the most widely used student outcome indicators in virtually every state, federal, 
and local educational indicator system (cf.. Governor's Commission on School 
Performance, 1989; Haertel, Haertel & Katzenmeyer, 1989; Oakes, 1986; OERI State 
Accountability Study Group, 1988). They are widely recognized as having the 
advantage of being reliable and objective. Georgia has in place a system of 
statewide student assessment which uses both criterion-referenced tests (CRTs), 
developed in Georgia and directed toward specific learning objectives that 
Georgia educators have agreed are essential for students' academic progress and 
success, and norm-referenced tests (NRTs). Two of the CRTs provide pass-fail 
scores: students must pass the third-grade CRT in order to be promoted to the 
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fourth-grade, and students must pass the Basic Skills Test given initially In the 
tenth-grade, in order to receive a high schod diploma. CRTs are given in grades 
1, 3, 6, S, and 10. There are also optional CRTs in grades 2 and 4. The NRTs used 
are the Iowa Tests of Basic Skills (TIBS) in grades 2, 4, and 7, and the Tests of 
Achievement and Proficiency (TAP) in grade 9 (QUE, 1988). Non-Asian minority 
students have achieved lower scores on standardized tests than non-minority 
students. The Coimdl of Chief State School Officers includes assessment of 
achievement outcomes for aU students, and for minorities, in its recommended 
system of educational outcome indicators (Blank, 1989). Achievement test scores 
for all students, and for minorities, are thus included as indicators of the degree 
to which the schools are educating elementary, middle, and secondary students. 

Students must be present in a school in order for the schoc' to educate 
them^ thus attendance is related both to school completion, as discussed above, 
and to the basic task of educating students. Student attendance or absenteeism 
rate is frequently assessed as an outcome indicator (cf., California Department of 
Education, 1989a, 1989b, 1989c; Gottfredson, 1985; North Carolina, 1989; OERI 
State Accountability Study Group, 1988) or recommended as an indicator 
(Governor's Commission, 1989; Haertel, Katzenmeyer, & Haertel, 1989; NCA 
Commission on Schools, 1985). Both overall student absenteeism, and minority 
absenteeism, were included as indicators for elementary, middle, and secondary 
schools. 

Student learning is also related to student discipline, thus rates of student 
suspension and expulsions are proposed as outcome indicators (Haertel, 
Katzenmeyer, and Haertel, 1989). The Greensboro Public School System (1989) 
has proposed inclusion of suspension and retention rates in its quality indicator 
system. Rates of suspension, expulsion, and retention overall, and for minority 
students, were included as indicators for elementary, middle, and secondary 
schools. 

At the request of reviewers in the Georgia Department of Education, the 
following indicators are also included: rates of student participation in math fairs 
and tournaments; rates of student enrollment and completion of Algebra I, 
Algebra U, and Geometry; and the percent of students nominated for, and 
selected for, the Governor's Honor's Program. 

Educating Elementary Students 

The stale of Virginia (McMillan, 1989) uses performance on the ITBS, student 
absenteeism, and percent of students passing the Literacy Passport (a 5th-grade 
competency test in Virginia) as indicators of the degree to which sdiools are 
educating elementary students. The Governor's Commission on School 
Performance (1989) has also reconunended that the percentage of students who pass 
competency tests be used as outcomes in Maryland's comprehensive educational 
evaluation system. Georgia has a competency test administered in the third-grade, 
hence the pass rate on this test, along with performance on the NRTs and CRTs, and 
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Student absenteeism, was included in the educational indicators. The OEKI (1988) 
also recommends consideration of student participation in arts and extracurricular 
activities in educational indicator systems. 

The middle schools to which elem^tary students matriculate can also provide 
feedback on the adequacy of the preparation of the students for nuddle school level 
work. This feedback is also included as an indicator of the effectiveness of 
elementary schools in educating students. 

Educating Middle School Students 

The state of Virginia (McMillan, 1989) uses performance on the ITBS and TAP, 
student absenteeism, percent of students passing the literacy Passport (an 8th-grade 
competency test in Virginia), and percent of students who have participated in at 
least one school sponsored extracurricular activity during the year as indicators of 
the degree to which schools are educating middle school students. The OERI (1988) 
also recommends consideration of student participation in arts and extracurricular 
activities in educational indicator systems. These indicators were Included for all 
students and for minority students. 

The high schools to which middle-school students matriculate can also 
provide feedback on the adequacy of preparation of students for high-school 
level work. This feedback is also included as an indicator of the effectiveness of 
elementary schools in educating students. 

Educating Secondary School Students 

The state of Virginia (McMillan, 1989) uses performance on the TAP, student 
absenteeism, percent of students who have taken keyboarding or typing, and percent 
of students who have participated in at least one school-sponsored extracturicular 
activity during the year as indicators of the degree to which schools are educating 
secondary school students. The percent of students who have taken keyboarding is 
not included as an indicator, as it is an indicator of very limited and specific 
educational outcomes. The percent of students who passed the CRT in the tenth 
grade was considered above, as an indicator related to school completion; here, 
average scores will be considered. The OERI (1988) also recommends consideration 
of student participation in arts and extracurricular activities in educational indicator 
systems. These indicators were included for all students, and minority students. 



Inaeasing Special Education Students' Living Skills & Opportunities 

Fewell and Vadasy (1987) stress the importance of specifically assessing the 
extent to which educational programs enhance the quality of life for special 
educational students and their families. The state of Virginia (McMillan, 1989) 
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uses the following indicators to assess the degree to which schools adequately 
contributed to Special Education Students' Living Skills & C^portunities: 
attendance and dropout rate of special education students, percent of handicapped 
students receiving an Honors Diploma, petcent of handicapped students 
attempting competency tests, percent of handicapped students passing competency 
tests, percent of handicapped students taking standardized tests, and rate of 
employment (or further education) of special education students after graduation. 
Evans and Brown (1986) suggest that tests of spedfic living skills, such as the ability 
to ties one's shoes or dial a telephone, tailored to individual students' 
Individualized Educational Plans, be used as outcome indicators to assess the 
effectiveness of programs for special education students. However, this seems 
infeasible in a state-wide testing program, and thus was not included here. 



REVIEW OF UTERATURE ON STANDARDS AND INDICATORS OF 
QUALITY FOR CHARACTERISTICS OF EFFECTIVE SCHOOLS 

As described in the body of this paper, the factors dted and described by 
Purkey and Smith (1983) and by Purkey and Degen (1985) encompass most of the 
findings of current studies on effective schools, and were used to organize 
development of standards and indicators of school characteristics for this project. 

These factors are: 

1. Autonomy of school-site management 

2. Instructional leadership 

3. Staff stability 

4. Parental involvement and support 

5. Schoolwide recc^nition of academic success 

6. Maximized learning time 

7. Collaborative planning and collegial relationships 

8. Sense of community 

9. Clear goals and high expectations 

10. Order and discipline 

Standards and indicators for each of these ten factors were developed 
through review of the pertinent professional literature. To accomplish the 
comprehensive review of the relevant professional literature upon which the 
comprehensive school characteristics indicator system was based, a computerized 
literature search (ERIC) was conducted. The sources identified in the literature 
search were augmented by a review of the educatbnal evaluation practices of 
other states and of professional papers presented at the latest meetings of the 
American Educational Research Association and the National Cotmcil on 
Measurement in Education. A synopsis of the results of review of the 
professional literature on each of the factors characteristic of effective schools 
listed above appears below. An integrated summary list of standards and 
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indicators for evaluation of educational outcomes and school characteristics 
appears in Appendix B of this paper. 

Autonomy of School-Site Management 

A number of studies indicate that effective schools are characterized by a 
considerable degree of autonomy of administrators and teachers in determining 
the exact means by which they will address school and student needs and 
problems. Purkey and Smith (1983) refer to this as autonomy of school-site 
management and view it as essential to the development of a school-spedfic 
culture. Based upon review of the professional literature, Austin (1979) also 
concluded that a sense of autonomy on the part of the principal was important to 
school effectiveness and that teachers feeling free to choose techniques and try 
new things was important to school effectiveness. The California Department of 
Education (1989a, 1989b, 1989c) includes assessment of the degree to which 
teachers feel encouraged to try new approaches or innovations in its Quality 
Review of schools. The Minnesota State Department of Education (Maruyama et 
al., 1989) also includes assessment of the autonomy of school site management 
and support for teaching innovations in its Minnesota Educational Effectiveness 
Project. 

Instructional Leadership 

Strong leadership has repeatedly been found to be associated with school 
effectiveness and is thought to be necessary to initiate and maintain the 
improvement process. Most studies And this leadership to be by the principal, 
although Purkey and Smith (1983) state that groups of teachers or other 
administrators could also provide leadership to the school. Based upon review 
of the professional literature, Austin (1979) and Lewis (1986) also concluded that 
the leadership of the principal was central to school effectiveness. Specifically, it 
was important for principals to exert power based upon educational expertise 
(not administrative expertise or other sources of power) and to be viewed by 
teachers and students as expert in a wide variety of areas pertaining to education 
(High and Achilles, 1986). Edmonds (1979) also reviewed the research literature 
on effective schools and concurred with the centrality of strong leadership on the 
part of the principal. Assessment of the leadership of tK rindpal in 
comprehensive school evaluations was advocated by Siroi^dk (1987) and by 
Bamburg and Andrews (1989). 

Staff Stability 

Sirotnik (1987) advocated the assessment of the stability of teachers and of 
the prindpalship in comprehensive school evaluations. I^urkey and Smith 
(1983, 1985) dte several studies which indicate that a stable staff appears necessary 
to promote and maintain success and that frequent staff turnovers are likely to 
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retard growth and impede change. Turnover rates for the principal and for 
teachers were included as indicators. 

Parental Involvement and Support 

Tne evidence concerning the impc^tance of parental involvement in 
schools appears mixed, vnth only a few studies finding it to be a major factor in 
student achievement (Wimpelberg et al., 1989). Purkey and Smith (1983, 1985) 
conclude that evidence supports the need for parents to be iitformed of school 
goals and student responsibilities, especially with regard to homework, 
discipline, and attendance. Based upon their reviews of the professional 
literature, Austin (1979) and Uwis (1986) concluded that positive parent-teacher 
relationships were important to school effectiveness. Parental involvement in 
decision making was suggested by Purkev and Degen (1985) as a means to foster 
parental support. 

In a state-wide empirical study in Minn«ota, Maruyama, et al. (1989) 
found parental support and involvement to be the "effectiveness" characteristic 
most highly correlated with elementary reading achievement. In a study of 16 
effective schools in the Chicago area, Wynne (1984) also found frequent 
communication with parents and parental involvement to be characteristic of 

these schools. Georgia's commitment to parental involvement is evidenced by 
the fact that efforts "to improve parent involvement" were a priority focus of the 
GDE State Strategic Planning Team for 1989 (Rogers, 1989, March). The California 
Department of Education (1989a, 1989b, 1989c) includes assessment of parental 
support and involvement in its Quality Review of schools, as does the 
Minnesota State Department of Education (Maruyaii a et al., 1989) in its 
Minnesota Educational Effectiveness Project. In particular, the Quality Review in 
California includes assessment of the degree to which parents are kept informed 
of the progress of their children, students' homework responsibilities, school 
rules and goals, and special activities of the schools, as well as the degree to 
which parents have regular opportunities to share their expectations regarding 
school programs. The Greensboro Public School System (1989) has proposed 
inclusion of parental PTA membership and attendance rates in its quality 
indicator system. 

Schoolwide Recognition of Academic Success 

Purkey and Smith (1983, p. 444) state: "A school's culture is partially 
reflected in its ceremonies, its symljols, and the accomplishments that it chooses 
to recognize officially. Schools that make a point of publicly honoring academic 
achievement and stressing its importance through the appropriate use of 
symbols, ceremonies and the like encourage students to adopt similar norms and 
values." They dte several stiidies supporting this conclusion. The California 
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Department of Education (1989a, 1989b, 1989c) includes assessment of the degree 
to which studente are recognized for good work in its Quality Review of schools. 

Maximized Learning Time 

In characterizing the factor which they refer to as maximized 
learning time, Purkey and Smith (1983) include some characteristics which 
address how instruction in an individual classroom is organized and 
managed. Such considerations are addressed in other chapters of this 
volume which concern evaluation of specific academic programs. 
However, Purkey and Smith also include some considerations that cross 
classroom or subject-area boundaries, indicating that the school 
environment as a whole is managed so that instructional time is 
maximized. They conclude that research supports the importance of class 
periods being free from interruptions by loudspeaker, messages from the 
counseling office, or disruptions from the hall or yard outside. The NCA 
Commission on Schools (1985) also advocates assessment of maximization 
of engaged learning time. The California Department of Education (1989a, 
1989b, 1989c) includes assessment of the de^ee to which the school 
environment is arranged to maximize learning time in its Quality Review 
of schools. In particular, the Quality Review in California includes 
assessment of whether announcements and disruptions interrupt class, and 
whether teachers begin instruction punctually and utilize the entire 
instructional period. 

Collaborative Planning and Collegial Relationships 

Eight studies which support the association of positive student outcomes 
with collegiality among teachers and administrators are dted by Purkey and 
Smith (1983), suggesting that a collegial atmosphere exerts a positive influence in 
several ways. It breaks down barriers between departments and between teachers 
and administrators, encourages intellectual sharing and consensus, promotes a 
feeling of unity among the staff, and allows the staff to work together. These 
studies also report the related finding that teachers and administrators working 
together is associated with positive student outcomes. Austin (1979) and Lewis 
(1986) also concluded that mutual respect, collegial relations, and collaborative 
work among teachers were important to school effectiveness. Maruyama et al. 
(1989) found collegiality to be weakly correlated with elementary reading 
achievement. Georgia's commitment to providing a supportive work climate to 
its teachers is evidenced by the fact that efforts "to improve work climate" for 
teachers was a priority focus of the GDE State Strategic Planning Team for 1989 
(Rogers, 1989, March). The NCA Commission on Schools (1985) advocates 
assessment of faculty perceptions of trust, security, and respect The California 
Department of Education (1989a, 1989b, 1989c) includes assessment of the degree 
to which staff members work cooperatively, the degree of respect among staff, 
and the degree to which teachers are recognized for good work in its Quality 
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Review of schools. The Minnesota State Department of Education (Maruyama, 
et al., 1989) includes assessment of (tie collegiality of the work atmosphere in its 
Minnesota Educational Effectiveness Project. 

Sense of Community 

Purkey and Simth (1983, p. 445) die two studies in support of the 
conclusion that: "There is persuasive evidence that community feeling, the 
sense of being a recognizable member of a community that is supportive and 
clearly perceived (by the staff and others), contributes to reduced alienation and 
increased achievement. There is also evidence that schools can create or build 
community by the appropriate use of ceremony, symbols, rules (i.e. dress code) 
and the like." Austin (1979) and Lewis (1986) also concluded that having teachers 
who were perceived as warm and responsive by students was important to 
school effectiveness. A similar, but less well defined construct, "school climate," 
is widely dted as assodated with school effectiveness (Kelley, 1981; Sagor, 1981; 
Sirotnik, 1987). In a multivariate analysis using data from 55,000 students 
collected over a three-day evaluation of instruction, Crantcn (1982) found the 
factor of school atmosphere to be the strongest predictor of student success. 
Maruyama et al. (1989) found school climate to be weakly correlated v«th 
elementary reading achievement. The NCA Commission on Schools (1985) 
advocates assessment of student and faculty perceptions of morale, trust, seciirity, 
and respect. The CaUfomia Department of Education (1989a, 1989b, 1989c) 
indudes assessment of school climate in its Quality Review of schools. In 
particular, the Quality Review in California indudes assessment of whether 
students feel the school environment is safe and supportive. The Minnesota 
State Department of Education (Maruyama, et al., 1989) i>^dudes assessment of 
the school climate in its Minnesota Educational Effectiveness Project, in 
particular whether the climate supports academic achievement for all students. 

Clear Goals and High Expectations 

Twelve studies supporting the conclusion that effective schools are 
characterized by dearly defined goals for students and high ^cpectations for 
student achievement on the part of botii students and staff are dted by Purkey 
and Smith (1983). They suggest that dearly defined goals help reduce student 
alienation, which they view as a common barrier to school effectiveness. Austin 
(1979) also conduded that high staff expectations for student performance were 
important to school effectiveness, as were positive self-concepts on the part of 
students. Edmonds (1979) and Lewis (1986) also reviewed the research literature 
on effective sdtools and concurred that higji expectatioi« for student 
performance were important to school effectiveness. Edmonds further 
conduded that sdiool effectiveness was assodated with teachers holding the 
belief that they could promote achievement even for disadvantaged children, 
rather than viewing the home and environmental influences in students' lives 
as insurmountable. Sirotnik (1987) also advocated assessment of teachers' 
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expectations for students in comprehensive school evaluations. The NCA 
Commission on Schools (1985) advocate assessment of the clarity of expectations 
for students. The California E)epartment of Education (1989a, 1^9b, 1989c) 
includes assessment of the clarity of goals, the degree to which teachers feel 
responsible for student learning, and the degree of high expectations for students 
in its Quality Review of schools. The expectation that all students complete all 
homework assignments, and that these are reviewed and returned in a timely 
manner, is used as an indicator in the California system. The Minnesota State 
Department of Education (Maruyama, et al., 1989) includes assessment of high 
expectations and the clarity of goals in its Minnesota Educational Effectiveness 
Project. At the request of reviewers in the Georgia Department of Education, the 
fact that course grading reflects only academic performance (rather than 
discipline or behavior) was included as an indicator in this section, as well as the 
percent of mathematics teachers who are certified to teach mathematics. 

Order and Discipline 

The existence of clear, reasonable rules that are fairly and consistently 
enforced can reduce behavior problems, wnich are often a barrier to learning; 
students cannot learn in an environment that is noisy, distracting or unsafe. 
Purkey and Smith (1983) dte a number of studies that suggest the importance of 
order and discipline for student achievement. The California Department of 
Education (1989a, 1989b, 1989c) includes assessment of the extent to which 
students feel safe, the consistency of enforcement of school rules, and the 
cleanliness and orderliness of the school environment in its Quality Review of 
schools. The Greensboro Public School System (1989) has proposed inclusion of 
the frequency of incidence of drug abuse in its quality indicator system- The 
Minnesota State Department of Education (Maruyama, et al., 1989) includes 
assessment of discipline and order in its Minnesota Educational Effectiveness 
Project. 
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APPENDIX B 

COMPREHENISVE LIST OF STANDARDS AND INDICATORS 
OF QUALITY FOR FHE EVALUATION OF 
EDUCATIONAL OUTCOMES AND SCHOOL CHARACTERISTICS 

Alt integrated summary list of standards and indicators for evaluation of 
school-based educational outcomes and school characteristics follows. Standards 
1 through 8 address educational outcomes, while Standards 9 through 18 address 
school characteristics. 

Educational Outcomes 

Improving School Completion Rate 

Standard 1: Students have acceptable rates of completion of their courses of 
study. 

a- The overall dropout rate (percent of f-tudents not returning to any 
school in the fall or who do not complete the year) is acceptable. 

b. The minority dropout rate (percent of minorities not returning to 
any school in the fall or who do not complete the year) is acceptable. 

c. The overall graduation rate is acceptable. 

d. The minority graduation rate is acceptable. 

e. The percent of students passing the Georgia Basic Skills Tes^ on first 
attempt (in the tenth grade) is acceptable. 

f. The percent of minorities passing the Georgia Basic Skills Test on 
Brst attempt (in the tenth grade) is acceptable. 

g. The percent of students scoring in the lowest quartile of the ITBS or 
TAP (depending on grade) is acceptable 

h. The percent of minorities scoring in the lowest quartOe of the ITBS 
or TAP (depending on grade) is acceptable. 



A Researdt-Based Attribute Stmcttire for School Accountability 



32 



Preparing students for post-secondarv school lifei 

Standard 2: The school contributes adequately to preparing students for 
continued schooling. 

a. The percent of students at the school receiving the Honors Diploma is 
acceptable, (for secondary schools) 

b. The percent of minorities at the school receiving the Honors Diploma is 
acceptable, (for secondary schools) 

c The percent of 11th and 12th grade students taking the SAT or ACT is 
acceptable, (for secondary schools) 

d. The percent of 11th and 12th grade minorities taking the SAT or ACT is 
acceptable, (for secondary schools) 

e. Students' scores on the SAT and ACT are acceptable, (for secondary 
schools) 

f. Minorities' scores on the SAT and ACT are acceptable, (for secondary 
schools) 

g. The percentage of students who meet state university entrance 
requirements is acceptable, (for secondary schools) 

h. The percentage of minorities who meet state university entrance 
requirements is acceptable, (for secondary schools) 

i. The number of National Merit Scholarship qualifiers, semi-fmalists, and 
finalists, or students who qualify for other scholarships or awards based on 
aptitude, is acceptable, (for secondary schools) 

j. The percent of students taking advanced placement or college level courses 

is acceptable, (for secondary schools) 
k. The percent of minorities taking advanced placement or college level 

courses is acceptable, (for secondary schools) 
1. The enrollment rate in advanced mathematics courses is acceptable, (for 
• secondary schools). 

m. The enrollment rate of minorities in advanced mathematics courses is 

acceptable, (for secondary schools) 
o. The enrollm. nt rate of females in advanced mathematics courses is 

acceptable, (for secondary schools) 
p. The enrollment rate in advanced science courses is acceptable, (for 

secondary schools) 



A Research-Based Attribute Structure for School Accountability 



q. The enroUment rate of minorities in advanced science courses is acceptable. 

(for secondary schools) 
r. The enroUment rate of females in advanced science courses is acceptable. 

(for secondary schools) 
s. The enrollment rate in foreign language courses is acceptable, (for 

secondary and middle schools) 
t. The minority enrollment rate in foreign language courses is acceptable, (for 

secondary and middle schools) 
u. The percent of students taking algebra is acceptable. (for middle schools) 
V. Percent of minority students taking algebra is acceptable, (for middle 

schools) 

w. The percent of female students taking algebra is acceptable, (for middle 
schools) 

X. The percent of students scoring in the upper quartile of the ITBS or TAP 

(depending on grade) is acceptable, (for all schools) 
y. The percent of minorities scoring in the up]>er quartile of the ITBS or TAP 

(depending on grade) is acceptable, (for ail schools) 
z. The percent of graduates who subsequently attend college is acceptable, (for 

secondary schools) 
aa. The percent of minority graduates who subsequently attend college is 

acceptable, (for secondary schools) 

Standard 3: The school contributes adequately to preparing students for work. 

a. The percent of vocational education students who complete their 
vocational education program is acceptable, (for middle and 
secondary schools) 

b. The percent of minority vocational education students who 
complete their vocational education program is acceptable, (for 
middle and secondary schools) 

c. The graduation rate of vocational education students is acceptable, 
(for secondary schools) 

d. The percent of students who have taken a vocational aptitude test or 
interest inventory is acceptable, (for middle and secondary schools) 
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e. The percent of vocational education students competing for district, 
regional, state and national awards is acceptable, (middle & 
secondary schools) 

f. The percent of graduating students not attending college who are 
employed full time in an area related to their preparation or are in 
the military within one year of graduation is acceptable, (using 
adjustment for local labor market conditions), (secondary schools) 

g. The percent of graduating minorities not attending college who are 
employed full time in an area related to their preparation or are in 
the military within one year of graduation is acceptable , (using 
adjustment for local labor market conditions), (secondary schools) 



Providing students with school experiences appropriate to their ag es. 
developmental levels and skill levels:* 

Standard 4: The school contributes to increasing special education students' 
living skills and opportunities. 

Indicators: 

a. The attendance rate of special education students is acceptable. 

b. The dropout rate of special education students is acceptable. 

c. The percent of students with hearing, speech, visual or orthopedic 
impairments, or learning disabilities who receive Honors Diplomas 
is acceptable. 

d. The percent of sp^jcial education students who attempt the third- 
grade competency test is acceptable. 

e. The percent of special education students who pass the third-grade 
competency test is acceptable. 

f. The percent of special education students who attempt the tenth- 
grade competency test is acceptable. 

g. The percent of special education students who pass the tenth-grade 
competency test is acceptable 



* Standards 4, 5, 6 & 7 were subsequently connbined into a single standard: 'The school 
effectively educates students in the grade-levels it serves." 
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h. The percent of special education students taking the ITBS or TAP 
(depending on grade) is acceptable. 

i. Of spedal education students who take the tests, the performance of 
special education students on the ITBS or TAP (depending on grade) 
is acceptable. 

j. The percent of special education students who are employed (or 

enrolled in further education) after graduation is acceptable, 
k. The average percent of students' lEP annual goals which are met by 

the end of the year is at or above the established level. 
1. The degree of movement of students to a less restrictive academic 

environment (e.g. the increase in percent of time spent in a regular 

classroom) is at or above the established level. 

Standard 5: The school adequately educates elementary school students. 
(Applies to elementary schools only). 

Indicators: 

a. Student performance on the ITBS in grades 2 and 4 is acceptable. 

b. Minority performance on the ITBS in grades 2 and 4 is acceptable. 

c. Student performance on the CRT in grades 1, 3, and 6 is acceptable. 

d. Minority performance on the CRT in grades 1, 3, and 6 is acceptable. 

e. Percent of students passing third-grade CRT is acceptable. 

f. Percent of minority students passing third-grade CRT is acceptable. 

g. The student absenteeism rate is acceptable- 

h. The minority absenteeism rate is acceptable. 

i. The teacher absenteeism rate is acceptable. 

j. The out-of-school suspension rate is acceptable. 

1. The out-of-school suspension rate for minorities is acceptable. 

m. The in-school suspension rate is acceptable. 

n. The in-schooI suspension rate for minorities is acceptable 

o. The expulsion rate is acceptable. 

p. The expulsion rate for minorities is acceptable. 

q. The student retention-in-grade rate is acceptable. 

r. The minority retcntion-in-grade rate is acceptable. 
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s. Feedback from the middle schooKs) to which the students matriculate 

indicates that students are adequately prepared, 
t. Rates of student participation in the arts and extracurricular activities is 

acceptable. 

Standard 6: The school adequately educates middle school students. 
(Applies to middle schools only). 

Indicators: 

a. Student performance on the ITBS in grade 7 and the TAP in grade 9 is 

acceptable. 

b. Minority performance on the ITBS in grade 7 and the TAP in grade 9 is 

acceptable. 

c. Student performance on the CRT in grade 8 is acceptable. 

d. Minority performance on the CRT in grade 8 is acceptable. 

e. The student absenteeism rate is acceptable. 

f. The minority absenteeism rale is acceptable. 

g. The teacher absenteeism rale is acceptable. 

h. The student retention-in-grade rate is acceptable. 

i. The minority retention-in-grade rate is acceptable, 
j. The expulsion rate is acceptable. 

k. The minority expulsion rale is acceptable. 

1. The out-of-school suspension rate is acceptable. 

m. The oul-of-school suspension rate for minorities is acceptable. 

n. The in-school suspension rale is acceptable. 

o. The in-school suspension rate for minorities is acceptable. 

p. Feedback from the secondary schooKs) to which the students matriculate 

indicates that students are adequately prepared, 
q. Rates of student participation in the arts and extracurricular activities is 

acceptable. 

Standard 7: The school adequately educates secondary school students. 
(Applies to secondary schools only). 

a. Student performance on the CRT in grade 10 is acceptable. 

b. Minority performance on the CRT in grade 10 is acceptable. 

c. The teacher absenteeism rate is acceptable. 
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d. The shidezit absenteeism rate is acceptable. 

e. The minority absenteeism rate is acceptable. 

t The student retention-in-grade rate is acceptable. 

g. Ihe minority retention-in-grade rate is acceptable. 

h. The student expulsion rate is acceptable. 

i. The minority expulsion rate is acceptable. 

j. The out-of-school suspension rate is acceptable. 

k. The out-of^chool suspension rate for minorities is acceptable. 

1. The in-school suspension rate is acceptable. 

m. The in-school suspension rate for minorities is acceptable. 

n. Rates of student participation in the arts and extracurricular 

activities is acceptable, 
o. The rate of student enrollment in Algebra I, Algebra Ih and 

Geometry is at or above the established level, 
p. The rate of student completion of Algebra I, Algebra 11, and 

Geometry is at or above the established level, 
q. The percent of students nominated for and selected for the 

Governor's Honor's Program is at or above the established level. 

Standard 8: The school adequately prepares the student for the life roles of learner, 
producer^ individual, citizen, consumer, and family member. 

Indicators: 

a. Students demonstrate positive and realistic self-concepts. 

b. Students respect and seek to understand themselves. 

c Students have positive attitudes toward school and learning. 

d. Students recognize the need for lifelong learning. 

e. Students value the arts, and respect creativity, beauty and excellence. 

f. Students value cultural diversity. 

g. Students possess personal value systems which emphasize consideration for 

others. 

h. Students understand the impact of the world community on individuals and 
sodety. 

i. Students are prepared to participate as citizens in our democratic society, 
j. Students understand that the quality of human life is enhanced by a 

harmonious relationship with the natural environment. 
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Students are prepared to function as intelligent consumers of goods and 
services. 

L Students are prepared to make responsible 
decisions. 

m. Students are prepared to function effectively as family members, 
n. Students seek to maintain sound physical and mental health. 

School Characteristics 

Standard 9: Teachers and adminislrators within the school have an appropriate 
degree of autonomy in addressing school and student needs and 
problems. 

a. Teachers have a large degree of professional autonomy and are 
encouraged to use their best professional judgment in carrying out 
the school's mission. 

b. Administrators report that they feel they have sufficient autonomy. 

c. The principal hires the teachers for the school. 

d. The school, rather than the district, organizes the inservice programs 
for school staff. 

e. The principal encourages teachers to try new ideas and techniques 
intended to promote achievement, and high goals and expectations. 

Standard 10: The school principal provides strong instructional leadership to the 
school. 

a. The principal demonstrates strong leadership qualities. 

b. Teachers and staff report that they feel the principal provides strong 
leadership to the school. 

c. Teachers and students perceive the principal to have wide expertise 
in the field of education. 

Standard 11: The school has a stable staff . 

a. The turnover rate for the prindpalship is acceptable. 

b. The tiuTiover rate for teadiers and staff is acceptable. 

Standard 12: Parents are involved in and support the school. 

a. Parents are aware of students' homework responsibilities. 

b. Parental membership and attendance of PTA meetings are acceptable. 

c. Parents indicate that they feel the school is doing a good job. 
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d Parents are kept informed of the progress of their children on an 
ongoing basis, throtigh newsletters, teacher and principal 
communiques, and conferences. 

e. Parents have regular opportunities to share their expectations 
regarding the school programs. 

f. Parents are kept informed of school rules and goals, and special 
activities of the schools. 

g. Parents support their children in completing homework and 
academic assignments. 

h. The number and types of parent volunteers in the school are 
acceptable. 

Standard 13: The school promotes school-wide recognition of academic 
achievement of students. 

a. Teachers report that they feel acadenuc achievement is recognized 
school-wide. 

b. Students report that they feel academic achievement is recognized 
school-wide. 

c. The school has an active honor society. 

d. Students are recognized for their citizenship and for academic 
achievements in assemblies, through special recognition by the 
principal, through award notices sent to parents, and by other public 
means. 

Standard 14. The school environment is organized to maximize learning time. 

a. The school has and follows a policy prohibiting loudspeaker 
announcements, except for emergencies or fire drills, during class 
periods. 

b. The school has and follows a policy prohibiting interruption of class 
periods for messages to students or teachers, except in cases of 
emergency. 

c. Noise from the halls or outside yard are not sufficient to disrupt 
instruction during class periods. 

d. Teachers begin instruction promptiy and maintain student 
involvement throughout the entire instructional period. 

Standard 15: Relationships between and among teachers and administrators is 
collegial in nature, and staff engage in collaborative planning. 

a. Teachers, staff, and administrators report that they feel the school 
atmosphere is collaborative. 

b. Teachers and administrators engage in collaborative planning. 

c. Teachers and staff work cooperatively in developing and carrying out 
schoolwide policies. 
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d. Teachers and staff are respected as professionals, and their individual 
strengths are recognized. 

e. Teachers and other staff members receive recognition that 
acknowledges excellence in teaching, curricular knowledge, special 
abilities to work with other staff or with students, and willingness to 
contribute extra time on student activities. 

f. Teachers frequently observe other teachers and provide feedback to 
then\. 

g. Teachers generally enjoy interacting with colleagues from their 
school. 

Standard 16: The students and staff at the school have a clear sense of belonging 
to a supportive community. 

a. Students and staff report that they feel the school environment is 
supportive and caring. 

b. Students and staff report that they have a sense of belonging to or 
identity with the school community. 

c Students and staff report that they perceive the school as a safe place 
lobe. 

d. The school climate supports academic goals. 

e. Students report that teachers take a strong interest in their 
individual academic performances. 

f. All students feel welcome and encouraged to participate in classroom 
learning activities. 

g. Teachers respect all of their students. 

Standard 17: The school culture presents clear goals and high expectations for 
students. 

a. Students report that they are aware of clear goals for their 
performance. 

b. Students have high expectations for their own performance. 

c. Teachers have high expectations for all students' performance. 

d. Students have positive self-concepts. 

e. Teachers believe that they can influence student achievement for all 
students, (rather than believing the influence of disadvantaged 
students' backgrounds to be insurmountable.) 

f. Parents, students and staff are all involved in setting standards for 
b'.-havior. 

g. All students are expected to complete all homework assignments. 

h. Assignments are reviewed and returned to students in a timely 
manner. 

i. Compared to other issues in the school, such as discipline, peer 
relations, or participation in extra-curricular activities, the staff and 
students think academic achievement is very impoitant. 
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Standard 18: Th<« school maintains an orderly and safe environment. 

a. Students and staff report that they feel safe. 

b. Students believe that the school's rtUes are fair and are consistently 
enforced. 

c. Staff believe that the school's rules are fa?r and are consirtently 
enforced 

d. The school eaviroiunent is clean and orderly. 

e. Violence and vandalism are rare occurrences at the school. 

f. The school does not have excessive incidence of drug use or sales. 

g. The school does not have excessive incidence of theft. 

h. The school does not have excessive incidence of cheating. 



A Researdv-Based Attribute Structure for School Accountability 



42 



APPENDIX C 

PROPOSED USE OF NUMERIC DATA FOR SELECTED INDICATORS OF 

QUALITY 

As stated in the body of this paper, once the Georgia CES is fully 
developed and in place, schools will be visited and reviewed once every five 
years. We have proposed that, when the CES is fully operational, the 
evaluation review vdll consider the status of many outcome indicators (such 
as graduation rates, dropout rates, attendance, and r Jtes of staff turnover) 
across the years since the last site visit, rather than limiting consideration to 
the outcomes of a single year. This would p; otect schools from receiving an 
unduly h?. 5h evaluation because they happen to be reviewed during a year 
when soiT". outcome is unusually low for them. Considering outcomes 
across all years since the last site review would also help schools keep an 
ongoing focus on striving for quality of outcomes and will maximize 
r^easurement reliability. During the developmental years of the CES, all 
schools will not have maintained five-year longitudinal records on all 
relevant student outcomes- Thus, the method proposed for using the 
longitudinal data had to accommodate cases where data for all years were not 
available. 

We have proposed that the data for a number of outcome indicators be 
treated in a way analogous to that described below for graduation rate. As the 
section below shows, if data on yearly graduation rate are available for less 
than three years, the indicator "overall graduation rate*' is simply 
operationally defined as average across the years for which data are available. 
If, however, the yearly graduation rate is available for three years or more, the 
indicator "overall graduation rate" is operationally defined by a formula 
which incorporates the change in the graduation rate across the years for 
which data are available. If the school's graduation rate has increased 
steadily, it will be credited with its highest rate over the span of years; a 
steadily decreasing graduation rate will result m the school being credited 
with its lowest graduation rate over the span of years. Miiumums and 
maximtmtis are also set to prevent the rate from being above 1.00 or below 
0.00. These limits would be needed only rarely, since, except in very unusual 
circumstances, the formula will yield a value within the range of rates 
achieved by the school over the span of years for which data are available. 

The next page of this appendix presents the operational definition of the 
indicator "overall graduation rate." As stated above, analogous operational 
definitions are proposed for a number of other indicators. 
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Indicator: The overall graduation rate meets or exceeds the established level. 

Operational cfefinitioa and data elements: Assessmem of this indicator requires the collection 
of two lands of data clenjcnts for the most reccndy completed school year and fw as many 
previous school years as possible, up to a maximum of five years. The kiiKis of data elements 
are (1) the total number of students who graduated from the school at any time during the 
regular school year (Numbw of Graduatesi) and (2) Ac Grade 12 enrollment during the school 
year (Senior Enidhnei^). These data are used to conq>ute a graduation rate for the school for 
each school year for which data are available (Gi). The yearly graduation rate equals the 
Number of Graduates divided by the Senior EnroUmenL The formula for the mdicator is 
based on the average of the graduation rales across all years for which data arc available. If 
data are available for at least three consecutive school years, dw indicaiOT adjusts the average 
graduation rate by an amount that depends on the change in the school's graduation rate oves 
Se span of years fOT which data arc available. If the school's graduation rate has mcrwsed 
over die span of years, die school will be credited widi diat improvement; if its grwiuanon rate 
has decreased. d» school's indicator value will be reduced to reflect its decreasing grwiuation 
rate. If die school's graduation rate has increased steadUy. it will be credited with its highest 
rate over die span of years; a steadily ctecreasing graduation rate wiU result m the school being 
credited widi its lowest graduation rate over die span of years. The formula will never let die 
indicator be larger dian 1.00 or smaller than zero. 

The indicator is operationally defined by die f^nula: 
Overall Graduation Rate 

= Min r I MU. [it(Oi) + ^^p).1.00l, I Maxlxi(Gj) + ^|J).0.0Ol] ,ifY23 



= :^I(Gi),ifY<3 
where: 

Gi 
Y 

Number of Graduates] 
Senior EnrDllmcnti 



( Number of GraduaicSj \ 
Senicx^EnroIlinoib / 

s the number of consecutive years for which data needed to 
compute Gi are available (Y^S) 

- the total number of students graduating during regular school 
Year i. including bodi fall and spring semester graduates. 

- the senior enrollment for Year i: 



P « die slope of die regression of Gj on Ycarj, for die past Y years 
(with die most recent year numbered " Y." and die most distant 
year numboed T), as derived from die formula: 

P=P (Oy/Ox) 

where:p ^ die correlation of Yeaq and Gi 

Oy s the standard deviation of Gi 

/ (Y-^iXY-l) 

= the standard deviation of Yearj = V 12 
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